Method and system for tracking an object

ABSTRACT

A method of tracking an object across a stream of images comprises determining a region of interest (ROI) bounding the object in an initial frame of an image stream. A HOG map is provided for the ROI by: dividing the ROI into an array of M×N cells, each cell comprising a plurality of image pixels; and determining a HOG for each of the cells. The HOG map is stored as indicative of the features of the object. Subsequent frames are acquired from the stream of images. The frames are scanned ROI by ROI to identify a candidate ROI having a HOG map best matching the stored HOG map features. If the match meets a threshold, the stored HOG map indicative of the features of the object is updated according to the HOG map for the best matching candidate ROI.

RELATED APPLICATIONS

The present application is a divisional of U.S. application Ser. No.15/426,413, filed on Feb. 7, 2017, which is a continuation-in-part ofPCT Application No. PCT/EP2016/063446 (Ref: FN-471-PCT) filed Jun. 13,2016.

FIELD

The present invention relates to a method and system for tracking anobject of interest across a stream of images.

BACKGROUND

US 2002/0102024, hereinafter Viola-Jones discloses a method fordetecting a region of interest (ROI) comprising an object such as a facewithin an acquired image, usually an image frame in a video stream. Inbrief, Viola-Jones first derives an integral image from the acquiredimage. Each element of the integral image is calculated as the sum ofintensities of all points above and to the left of the point in theimage. The total intensity of any sub-window in an image can then bederived by subtracting the integral image value for the top left pointof the sub-window from the integral image value for the bottom rightpoint of the sub-window. Intensities for adjacent sub-windows can beefficiently compared using particular combinations of integral imagevalues from points of the sub-windows.

Object detectors based on Viola-Jones, use a chain (cascade) of, forexample, 32 pre-trained classifiers based on rectangular (andincreasingly refined) Haar features with the integral image by applyingthe classifiers to a candidate sub-window within the integral image. Fora complete analysis of a scan area within an acquired image, thissub-window is shifted incrementally across the integral image until thescan area has been covered.

It will be appreciated that applying Viola-Jones analysis to everyportion of an image for every size of object to be detected can still bequite processor intensive and this could prevent a system operatingquickly enough to detect and track an object across a stream of imagesin real time.

Thus, many improvements of this basic technique have been developed. Forexample, PCT Application WO2008/018887 (Ref: FN-143), the disclosure ofwhich is incorporated by reference, discloses an image processingapparatus for tracking faces in an image stream. Each acquired image ofthe stream is sub-sampled at a specified resolution to provide asub-sampled image. Fixed size face detection is applied to at least aportion of the integral image to provide a set of candidate faceregions. Responsive to the set of candidate face regions produced andany previously detected candidate face regions, the resolution isadjusted for sub-sampling a subsequent acquired image.

There remains a need however for a more efficient mechanism for trackingone or more objects across a stream of images.

SUMMARY

According to the present disclosure there is provided a method oftracking an object.

In a further aspect there is provided a system for tracking an object.

Embodiments of the invention can provide simple and hardware friendlyobject tracking.

In some embodiments, an original ROI that contains an object of interest(target) that is to be tracked is automatically detected in an initialframe within a sequence of frames using a first detection mechanism.

Alternatively, a ROI could be defined through interaction with anapplication in which a user defining a box with a pointer such as amouse or on a touchscreen. In this case, the user could interact with apreview image stream and once the ROI is defined, the frame in which ithas been defined can be taken as the initial frame.

In some embodiments, a feed forward neural network (NN) is first trainedby determining the weights for the connections within the neural networkfor the ROI based upon the features of the ROI within the initial frame.

Each subsequent frame, is scanned with a sliding window around thelocation of the last detected ROI, with the trained NN being used tolook for a ROI with the best match based on the existing weights.

The location of the ROI with the best match is reported and the NN isretrained with the features for the newly found ROI with the best match.

In this way the method can adapt to small changes of the tracked objectfrom frame to frame (e.g. rotation, illumination), or changes in the ROIbackground area around the object.

NN retraining can be done taking into account only the ROI in thecurrent frame, or a history of the tracked ROI from the last X frames ina video sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the invention will now be described, by way of example,with reference to the accompanying drawings, in which:

FIG. 1 shows schematically an object tracking system according to anembodiment of the present invention;

FIG. 2 illustrates generally how a histogram of gradients (HOG) isextracted from an image and how this is used to generate correspondingHOG maps for use in the embodiment of FIG. 1;

FIG. 3 shows a HOG bin configuration employed in an embodiment of thepresent invention;

FIG. 4 shows a neuron architecture employed within the neural network ofFIG. 1;

FIG. 5 an exemplary Sigmoid activation function and a piecewise linearapproximation of such a function employed within the embodiment of FIG.1; and

FIG. 6 illustrates an object tracking system according to a secondembodiment of the present invention.

DESCRIPTION OF THE EMBODIMENTS

FIG. 1 shows an object tracking system 10 according to an embodiment asdisclosed herein.

The system 10 is typically implemented within an image acquisitiondevice which obtains a video sequence comprising a series of frames froma source. Typically, the system is implemented in a device such as adigital camera, smart phone, tablet or general purpose computerincluding a camera and the sequence of images is provided directly. Inalternative embodiments, the system 10 could be implemented in a networkdevice which obtains the video sequence from a separate source deviceacross the network.

In brief, the system 10 includes a first mechanism 12 for determining aregion of interest (ROI) within an image. In one embodiment, themechanism 12 comprises a pre-trained classifier based on Haar featuressuch as disclosed in PCT Application WO2008/018887 (Ref: FN-143)referred to above. When such a classifier is applied to an image, it canidentify one or more ROI within the image. Depending on the types ofclassifier employed, the ROI can vary in size, orientation and content.In any case, once a ROI is identified in a frame of the image sequence,this frame is treated as an initial frame.

In an alternative embodiment, the system 10 is implemented within aninteractive device where the first mechanism 12 comprises an interactiveapplication arranged to provide a preview display of a video sequence.The user can interact with the preview to manually define one or moreROI within the preview. An image in which such a ROI is defined can beregarded as an initial frame for the purposes of the presentdescription.

While two examples of first mechanism 12 have been mentioned above, itwill be appreciated than any suitable mechanism for identifying a ROIwithin an initial frame can be employed.

Also, while the present embodiment is described in terms of processing asingle ROI within an image, it will be appreciated that the invention isequally applicable to detecting and tracking more than one ROI within animage sequence.

In any case, for any given ROI in an initial frame which has beenidentified by the first mechanism 12 or as will be explained later, forany candidate ROI from a subsequent frame which is to be examined, thefeatures of the ROI are extracted by a feature extraction block 16.

In the present embodiment, only one type of feature is used: Histogramof Oriented Gradients (HOG). Navneet Dalal and Bill Triggs, “Histogramsof Oriented Gradients for Human Detection”, IEEE Computer SocietyConference on Computer Vision and Pattern Recognition, vol 1, page886-893, 25-26 Jun. 2005; and Navneet Dalal “Finding People in Imagesand Videos” PhD Thesis, L'institut National Polytechnique De Grenoble,17 Jul. 2006, disclose HOG descriptors for use in image processingincluding for object detection and/or classification.

Note that in typical applications based on HOG classifiers, theclassifiers need to be trained a priori and this can be an involvedprocess. No such a priori training is required in the presentembodiments.

Referring to FIG. 2, when employing HOG, an image is divided into anarray of cells.

In the present example, 4×4 cells are used, however the size of cellcould vary and need not necessarily be square. The cells can abut,overlap or be spaced apart in the x and/or the y direction within animage and in the present application, cells overlap by 50% in each ofthe x and y directions.

A horizontal gradient, gx and a vertical gradient gy are determined foreach pixel of a cell and an orientation of the gradient is thendetermined. For images with 8 bit pixels, the values of gx and gy can beup to 255 i.e. the difference between pixel values on either side of agiven pixel within an image, the magnitude of the gradient, could be upto 255.

The range of orientations from 0 to 360° is divided into a number ofbins, q, and the orientation for each pixel is mapped to a bin. A HOGfor a cell can comprise a simple count of the number of instances oforientations of pixels found in each bin.

Some HOG generators are based on calculating the orientation of thegradient as

$\tan^{- 1}\frac{g\; y}{g\; x}$in order to determine the appropriate bin for a pixel gradient. However,it will be seen that performing this calculation for each pixel could beprocessor intensive.

U.S. Patent Application No. 62/085,346 filed 28 Nov. 2014 (Ref:FN-398-US), the disclosure of which is incorporated by reference,discloses a method for producing a HOG based on allocating a gradientfor a pixel to one of a plurality of sectors, each sector extendingthrough a range of orientation angles and at least some of said sectorsbeing divided from adjacent sectors along lines including gx=2^(n).gy.One or more sectors can be associated with a bin; and each instance of apixel gradient of a cell associated with a bin can be counted to providea HOG for the cell. This technique therefore enables a range oforientations to be divided into approximately equal sectors and toallocate gradients to those sectors without requiring a trigonometricfunction.

FIG. 2 shows a range of orientations divided into 24 sectors numberedfrom 0-23 divided along these lines. In this case, each sector can beaddressed with a 5 bit mask with the criteria for each bit of the maskbeginning at the most significant bit as follows:

1) gx>0;

2) gy>0;

3) |gx|>|gy|;

4) |gx|>2|gy| OR |gy|>2|gx|;

5) |gx|>4|gy| OR |gy|>4|gx|

As illustrated, the values required for a gradient to lie in sector 6,for example, are 01011.

Referring now to FIG. 3, in the embodiment, the 24 sectors are combinedinto q=6 bins as follows:

Bin 0: 5+6+17+18

Bin 1: 15+16+3+4

Bin 2: 1+2+13+14

Bin 3: 0+23+11+12

Bin 4: 7+8+19+20

Bin 5: 9+10+21+22

Thus for example, a pixel having a gradient falling in any of sectors 5,6, 17 or 18 would be allocated to bin 0.

FIG. 3 also shows a typical HOG for a cell where the y axis (Σmagnitude) indicates the count for each bin.

It will be appreciated that the above arrangement of bins is forexemplary purposes only and that q may vary from implementation toimplementation; that bin sizes can be uniform or non-uniform; and thatbins can span continuous or non-continuous ranges of orientations.

Referring back to FIG. 2, in the embodiment, the HOG generator 16produces q=6 HOG maps for each scanned area of an image based on theabove defined bins, the scanned area being divided into M×N cells. Eachmap therefore comprises M×N magnitudes, each magnitude corresponding tothe respective bin count for a corresponding cell of the scan area.

When using a simple count for HOG bins, for a 4×4 cell, the magnitudesfor each bin will range from 0-16 and so for larger cells this rangewill be greater and so the HOG map pixels need to be scaled accordingly.

In other applications, a magnitude for each pixel gradient can also betaken into account when generating a HOG for a cell. As disclosed inU.S. Patent Application No. 62/085,346, a magnitude of a gradient for apixel can be calculated as:mag=√gx ² +gy ²or approximated as follows:mag=|gx|+|gy|−min(gx,gy)/2

In either case, the value for each bin in a HOG can comprise the sum ofmagnitudes of gradients for pixels with orientations in that bin. Usingthis approach takes into account the weight of pixel gradients in agiven direction, but it also means that for a 4×4 cell of an 8 bitimage, the value for a bin might range from 0 to 16×255. As will beappreciated from the description below, it can be useful for the binvalues for such a HOG map to be normalized and so in some embodiments,once calculated, bin values can normalized so that values for a HOG maprange from 0 to 255 by scaling the initially calculated bin values asrequired.

U.S. Patent Application No. 62/085,346 referred to above discloses howwhen scanning across a scan area of an input image, a HOG generator canproduce 1 HOG/clock cycle. Thus, for an area of an image to be scanned,HOG maps can be produced in close to the same number of clock cycles aspixels or cells within the scan area.

It will be appreciated nonetheless that embodiments of the presentinvention can use either the advantageous HOG generator described inU.S. Patent Application No. 62/085,346 or any other suitable HOGgenerator to extract the necessary HOG features from a ROI within animage.

In the embodiment of FIG. 1, the features of an identified ROI withinthe initial frame i.e. the HOG map values, are used within a secondmechanism for tracking the object within the ROI detected within theinitial frame through subsequent frames of the video sequence. In theembodiment, the features are used as weights for the neurons of a feedforward neural network (NN) 14 as will be described in more detailbelow.

Referring now to FIG. 4 which shows the structure of the neural network14. The network has 3 layers of neurons with Layer 1 comprising M×N×qneurons, where q=6, the number of bins in each HOG. Thus, each group ofq neurons corresponds to the set of bin values for a given cell of thescan area.

For an initial frame, each HOG map value for the detected ROI(normalized if required) is used as the respective weight w_(j,i) for acorresponding neuron of Layer 1.

Layer 2 comprises M×N neurons at which the outputs of a corresponding 6neurons from Layer 1 are summed and fed through an activation functionf( ).

In the embodiment, a sigmoid activation function is employed and again,rather than require a trigonometric function or exponentiation to beperformed at the neurons of Layer 2, a piece-wise linear approximationof the sigmoid function is used.

Referring to FIG. 5, the piece-wise approximation is of the followingform:

${f\left( y_{i} \right)} = {\quad\left\{ {\quad\begin{matrix}{0,{{{if}\mspace{14mu} y_{i}} < a}} \\{{{0.0625 \times y_{i}} + 0.25},{{{if}\mspace{14mu} y_{i}} \in \left\lbrack {a,b} \right)}} \\{{{0.1875 \times y_{i}} + 0.5},{{{if}\mspace{14mu} y_{i}} \in \left\lbrack {b,c} \right)}} \\{{{0.0625 \times y_{i}} + 0.75},{{{if}\mspace{14mu} y_{i}} \in \left\lbrack {c,d} \right)}} \\{1,{{{if}\mspace{14mu} y_{i}} \geq d}}\end{matrix}} \right.}$

Here y_(i)=Σ_(j=0) ^(q−1)y_(j,i) where y_(j,i) comprises the product ofa weight w_(j,i) and a bin value b_(j,i) for a neuron in Layer 1. Layer2 has neurons with q=6 inputs that combine the q corresponding bins fromeach HOG cell into a single output.

In the example, the output of each neuron of Layer 2 is scaled from 0 to1, however, it will be appreciated that any suitable scale could beemployed. Similarly, the values a, b, c and d are scaled according tothe size of HOG cell.

Layer 3 comprise a single neuron summing the outputs of the M×N Layer 2neurons to provide a single output value (score). The output value istherefore proportional to the level of match between the HOG map valuesused for the weights of the Layer 1 neurons and the HOG map valuesprovided for a candidate ROI.

The nodes of Layers 2 and 3 may be weighted as described in more detailbelow.

A particularity of the neural network 14 is that is not fully connectedi.e. the inter-layer connections, especially those from the HOGs maps tothe neurons of Layer 1 are kept local. This reduces the complexity ofthe network and the computational cost.

Again, the above described arrangement of neural network 14 is providedfor exemplary purposes only and it will be appreciated that in variantsof the described embodiments, the number of neurons, their respectivefunctions and their interconnections may vary.

For example, neurons in Layer 1 might be multi-input neurons combininginformation from multiple bins.

In other variants, weights can be determined not only for nodes of Layer1, but for each layer of the neural network 14. Thus, the weights forLayer 1 can be calculated as above, but in this case, each neuron ofLayer 1 would also implement a sigmoid activation function.

After determining the weights for the Layer 1 based on the initial framecontaining a ROI, the activation function outputs of the Layer 1 nodesfor a given cell would then be combined and used as the weights forLayer 2 nodes. These weights could then be fed directly as outputs andcombined to act as the weight for the Layer 3 node.

The output of the Layer 1 nodes resulting from the combination of a HOGmap for a candidate ROI and the Layer 1 weights could then be fed as aninput to Layer 2 nodes, where it could be normalized and multiplied bythe Layer 2 weights; and subsequently, the output of the Layer 2 nodesresulting from the combination of the Layer 1 outputs and the Layer 2weights could then be normalized and fed as an input to the Layer 3node. In Layer 3, the single input could be multiplied by a singleweight to provide an output score.

Again, the weights for the Layer 1, Layer 2 and Layer 3 nodes for a bestmatch candidate ROI would be used for determining the weights forscanning a subsequent frame and updated in the same manner as describedabove.

Referring back to FIG. 1, for the initial frame, the HOG maps(normalized if required) for an initial ROI containing an object ofinterest (target) are used to generate the initial weights for theneurons of Layer 1.

Then, subsequent frames of the video sequence, are scanned in thevicinity of the location of the ROI from the immediately preceding frameor a function of the location of the ROI from the immediately precedingframes. So for example, if significant frame-to-frame movement isdetected a scan area around a previously detected ROI might need to berelatively large to capture all suitable potential candidate ROI,whereas for more static content, the scan area could be more closelytied to the previously detected ROI.

Note that the scan area need not be symmetric about a previouslydetection ROI and where frame-to-frame motion indicates movement of asubject in a given direction, the scan area could be definedaccordingly.

Indeed, in extremis, inter-frame movement might be such that a scan areafor a subsequent image might not completely bound a ROI detected in aprevious image—nonetheless, the scan area will always need to be atleast as large as a ROI.

In this respect, PCT Application WO2014/146983 (Ref: FN-389), thedisclosure of which is incorporated by reference, discloses an efficientmechanism for providing a map indicating inter-frame movement in videosequence and such a map where available could be employed to define theextent and location of a scan area within a subsequent frame of a videosequence.

In any case, the scan area of a subsequent frame is scanned ROI by ROIwith scanner 18 and HOG features are extracted by the HOG generator 16for each candidate ROI and normalized if required so that they have thesame range as for the weights of the neurons of Layer 1. Note thereforethat ROI from subsequent frames need to be spatially scaled to provideHOG maps of the same size M×N as the HOG map for the ROI from theinitial frame. Typically, the ROI for subsequent frames would be thesame size as the ROI for the initial frame.

The combination of candidate ROI HOG features from a scan area of asubsequent frame and the existing weights are fed through the neuralnetwork 14 to produce a score. The best score for a candidate ROI in agiven frame is noted by a scorekeeper 20. If the best score is above athreshold, the ROI producing that score is signaled as the new locationfor the target originally identified in the ROI from the initial frame.If the best score is not above a threshold, the system 10 might revertback to the first mechanism 12 to attempt to reacquire an initial ROI asbefore. In particular, this might occur where a target moves towards oraway from an acquisition system to the extent that its scale makestracking the object difficult. Nonetheless, as will be seen from thedescription below, as the features defining a target change over thecourse of a sequence tracking the target, the weights of the neuralnetwork 14 will also adapt so that even if the target appearance changesdramatically over the course of a sequence, it can still be tracked bythe system 10.

In any case, if a best score above a threshold is produced by acandidate ROI, then the HOG maps produced by the HOG generator 16 forthe candidate ROI are captured for use as new weights 22 within Layer 1of the neural network 14.

These weights can be used directly or, as in the embodiment illustratedin FIG. 1, weights 22 from corresponding cells of the HOG map for themost recent best matching ROI can be combined with the weights 24 forthe best scoring candidate ROI for the previous X frames, for example,using an averaging function 26, to generate the weights for the neuronsof Layer 1 and possibly Layer 2 and/or Layer 3 to be used in inspectingthe scan area of the subsequent image in the video sequence. Typicalvalues for X might be from 1 to 8 and it is thought that values higherthan 8 might be counterproductive, especially in a dynamic scene.

It will also be appreciated that functions other than the averagefunction 26 can be used and, for example, a weighted average favoringthe weights for best matching ROI from more recently acquired frames, orpossible a median based function could also be used instead of theaverage function 26.

It has been found that the above described embodiment performs quitewell in the face of: Illumination Variation (IV)—where the illuminationin the target region is significantly changed; Scale Variation(SV)—where the ratio of boxes bounding a target in the initial frame andthe target in a current frame is out of the range [1/ts, ts], ts >1(ts=2); Motion Blur (MB)—where the target is blurred due to the motionof target or camera; In-Plane Rotation (IRP)—where the target rotates inthe image plane; and Out-of-Plane Rotation (OPR)—wherein the targetrotates out of the image plane.

The embodiment performs well with mid-range performance in the face of:Background Clutters (BC)—where the background near the target has thesimilar color or texture as the target; and Low Resolution (LR)—wherethe number of pixels inside the ROI is less than tr (tr=400).

It is nonetheless acknowledged that the embodiment may not perform verywell in the face of: Occlusion (OCC)—where the target is partially orfully occluded; Deformation (DEF)—non-rigid object deformation; FastMotion (FM)—where the motion of the ground truth is larger than tmpixels (tm=20); and Out-of-View (OV)—where some portion of the targetleaves the view. The performance drop in these last 4 categories is dueto the fact that the above described simple embodiment doesn't include amechanism for handling sudden occlusion or an out of view subject,however, it will be appreciated that where target tracking is lost bythe loop centered on the neural network 14, the system 10 can alwaysrevert back to the first mechanism 12 to reacquire a target and so inthis respect, it need not perform any worse than a system based on thefirst mechanism alone.

Precision can be defined as a ratio of frames whose tracked ROI outputis within a given threshold from a box actually bounding a target,measured by the center distance between ROI and the bounding box.

In terms of precision, the embodiment performs well with mid-rangeperformance in the face of: IV, SV, IPR, OPR, LR; and not very well withlow performance in the face of: OCC, DEF, FM, OV. Again, this lowperformance for the last 4 categories is due to the fact that the abovedescribed simple embodiment doesn't have a good support for have asuitable mechanism for handling sudden occlusion or an out of viewsubject.

Note that both the first mechanism 12 and the feature extractor 16 canoperate on a grayscale image only and thus only a luminance channel forYCC, LAB or YTUV format images might be required by the system 10.

The above described embodiment has been described in terms ofexhaustively scanning in both an x and a y direction across a scan areaof an image to detect a best matching ROI from within the scan area. Itwill be appreciated however, that this involves O(L²) operations, whereL is the width or height of the scan area. In alternative embodiments,the scanning path across the scan area could be guided by the scoreproduced by successively scanned ROI. Thus, if movement in one directionimproves the score more than movement in another direction, the scanpath might follow the faster increasing score to reduce the overallnumber of operations required to identify the best match ROI.

The illustrated embodiments have been described as being based on arectangular ROI and a rectangular scan area, however, it will beappreciated that in other embodiments a non-rectangular ROI andnon-rectangular scan areas could be employed.

What will be appreciated however, is that in any implementation, thesecond mechanism 14, which need not be pre-trained, can be immediatelyprovided with the HOG map features it needs to begin tracking an objectby the pre-trained first mechanism 12.

Referring now to FIG. 6, in a variation of the above describedembodiments, tracking is based on feature maps generated by a neuralnetwork rather than specifically HOG maps.

As disclosed in David Held, Sebastian Thrun, Silvio Savarese, “Learningto Track at 100 FPS with Deep Regression Networks,” in EuropeanConference on Computer Vision, ECCV, 2016, it is known in objectdetection to employ a neural network, in this case CaffeNet, comprisinga series of feature extracting convolution layers followed by a seriesof feature classification fully connected layers. Some such networkshowever, can involve millions of parameters and so it is not feasible toimplement these in portable electronic devices such as smartphones.

Attempts to rationalize the implementation of such networks include S.Ren, K. He, R. Girshick and J. Sun, “Faster R-CNN: Towards Real-TimeObject Detection with Region Proposal Networks,” in Advances in NeuralInformation Processing Systems, NIPS, 2015 where the network comprises aRegion Proposal Network (RPN) and Region Classification Network (RCN).

In the RPN, input images are processed by a feature extractor, forexample, VGG16, GoogleNet (see C. Szegedy, W. Liu, Y. Jia, P. Sermanet,S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke and A. Rabinovich, “GoingDeeper with Convolutions,” in Computer Vision and Pattern Recognition,CVPR, 2015.), ResNet-100, PVANet, and feature maps are used to predict anumber of N (e.g. 200, 300) class-agnostic ROIs, each ROI beingindicated by a location, scale and aspect ratio.

In the RCN part, the feature maps for the ROI generated by the RPN arefed to the classifier, in order to predict a class and class-specificbox refinement for each proposed ROI.

Most computational cost is caused by the RCN part, due to the largenumber of ROIs to be classified. For instance, for Faster R-CNNalgorithm with GoogleNet as RPN, 800×600 input image size and 300 ROIsproposed by the RPN, the entire computational power is 36 millionmultiplications for each frame, from which, 17 million multiplicationsare required by the RCN network. If the maximum number of ROIs proposedper frame is 500, then there are 41 million multiplications/frame fromwhich, 22 million are executed in the RCN.

In the approach of FIG. 6, RPN 12-A followed by RCN 12-B is performeduntil a ROI containing an object of a given class to be tracked islocated as in the case of Faster R-CNN. As in the case of Faster R-CNN,each of these networks 12-A and 12-B can be trained offline, possiblyusing techniques such as disclosed in U.S. patent application Ser. No.15/413,312 (Ref: FN-606-US) to augment a training set of classifiedimages, with their parameters pre-stored and accessible to the tracker.

Once such an object is located, rather than continue to use RCN 12-B, afeature map produced by a layer Y of the RPN is chosen to provide theweight values for the feed forward neural network 14′. Note thattypically, as well as convolutional layers, the RPN may include someform of pooling or striding, and so the size of the feature maps willtend to reduce from layer to layer. The layer Y is chosen early enoughin the RPN layer structure to have sufficient features and thusgranularity to be useful, but not so early that the number of parameterswhich would then be required by the feed forward neural network 14′would be too great. For a typical 5 layer RPN 12-A, the feature map fromlayers 3 or 4 is likely to be most useful.

For subsequent frames, the RPN produces feature maps for a set of ROIaround the current ROI determined for the previous frame and drawn fromthe same layer Y output as for the current ROI. Indeed, the RPN can betrained to determine the search zone in subsequent frames around acurrent ROI. The feature maps for the set of ROI are fed through to thefeed forward neural network 14′ where they are combined with thepreviously stored weights and the results from each ROI are used tochoose the best match 20 in the current frame as in the approach of FIG.1.

Now the feature map for the best match ROI can be combined with theweights history for the previous X frames through which the object hasbeen tracked, for example using an averaging function 26, as before, toupdate the weights for the feed forward neural network 14′.

If at any stage, this tracking fails to track the object originallyidentified by the RCN 12-B satisfactorily, processing reverts to thefinal layers of the RPN 12-A and the RCN 12-B to attempt to locate a ROIfor a class of interest ab initio. So for example, if the RCN 12-B weretrained (offline) to detect 3 classes of people: standing, sitting andwalking, if a person standing were initially detected, tracking mightfail when they sat down and in this case, when re-enabled, the RCN 12-Bwould then likely locate the person sitting down, re-initialise the feedforward neural network 14′ with the feature map for this ROI and thencontinue tracking as before. Note that in this case, the convolutionallayer Y chosen to produce the feature map for the weights for the feedforward network 14′ need not be the same as for other classes.

In some embodiments, the RCN 12-B, is enabled once every Z frames aswell as if there is no valid match suggested by the tracker 14′-22.Newly detected objects are then used to reinitialize the feed forwardneural network 14′ with the weights (feature maps) for any newlydetected object (either in place of or in addition to tracking theoriginal object, where required).

By comparison to using features computed from a more deterministicfeature extractor such as, Histogram of Oriented Gradients HOG, Haar,Census, LBP, Integral Image, the feature maps produced by the RPN 12-Aare learned during training along with the RCN 12-B using a testdataset, and so they can be better suited and more robust for theclasses of object being detected/tracked by the system of FIG. 6.

The invention claimed is:
 1. A method of tracking an object across a stream of images comprising: a) passing an initial frame of said stream of images through a first neural network comprising a plurality of convolutional layers, each convolutional layer producing at least one feature map; b) providing a feature map output from said first neural network to a second multi-classifier neural network comprising at least one fully-connected layer to determine a region of interest (ROI) bounding an object of a given class in said initial frame; c) storing a feature map output from a first convolutional layer of said first neural network corresponding to said ROI as weights for at least a first layer of neurons of a third multi-layer neural network; d) acquiring a subsequent frame from said stream of images; e) scanning at least a portion of said frame ROI by ROI using said first neural network to produce respective feature maps from said first convolutional layer for each ROI; f) providing said feature maps to said third multi-layer neural network to provide an output proportional to the level of match between the feature map values used for the weights of the first layer of neurons and the feature map values provided for a candidate ROI to identify a candidate ROI having a feature map best matching the stored feature map; g) responsive to said match meeting a threshold, updating the stored feature map indicative of the features of said object according to the feature map for the best matching candidate ROI including updating the weights for said neurons according to the feature map for the best matching candidate ROI from said subsequent frame; and h) repeating steps d) to g) until said match fails to meet said threshold.
 2. A method according to claim 1 wherein each of said first and second neural networks are trained offline.
 3. A method according to claim 1 wherein at least one layer of said third neural network comprises a plurality of neurons performing a sigmoid activation function on the outputs of neurons from said first layer.
 4. A method according to claim 3 wherein a third layer of said third neural network comprises at least a neuron aggregating the outputs of the neurons of a second layer.
 5. A method according to claim 1 wherein said neural network is a forward connected neural network.
 6. A method according to claim 3 comprising providing a piecewise linear approximation of said sigmoid function.
 7. A method according to claim 1 wherein said scanning comprises scanning across a scan area of a subsequent frame, ROI-by-ROI, said scan area being a function of said best matching candidate ROI from a previous frame and inter-frame movement between a previous frame and said subsequent frame.
 8. A method according to claim 1 further comprising periodically performing steps a) to c).
 9. A method according to claim 1 comprising only performing steps a) to c) in response to said best match not exceeding said threshold.
 10. A method according to claim 1 wherein said first convolutional layer is an intermediate layer of said first neural network.
 11. An image processing system including digital logic for implementing the method of claim
 1. 